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Abstract 



Privacy policies often place requirements on the purposes for which a governed entity may use 
personal information. For example, regulations, such as HIPAA, require that hospital employees use 
medical information for only certain purposes, such as treatment. Thus, using formal or automated 
^ I methods for enforcing privacy policies requires a semantics of purpose requirements to determine 

^ ■ whether an action is for a purpose or not. We provide such a semantics using a formalism based 

- - - on planning. We model planning using a modified version of Markov Decision Processes, which 

exclude redundant actions for a formal definition of redundant. We use the model to formalize 
when a sequence of actions is only for or not for a purpose. This semantics enables us to provide 
an algorithm for automating auditing, and to describe formally and compare rigorously previous 
enforcement methods. 
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1 Introduction 



Purpose is a key concept for privacy policies. For example, the European Union requires that |The95j : 

Member States shall provide that personal data must be [. . .] collected for specified, 
explicit and legitimate purposes and not further processed in a way incompatible with 
those purposes. 

The United States also has laws placing purpose requirements on information in some domains such 
as HIPAA |OfF03] for medical information and the Gramm-Leach-Bliley Act jUnilO] for financial 
records. These laws and best practices motivate organizations to discuss in their privacy policies 
the purposes for which they will use information. 

Some privacy policies warn users that the policy provider may use certain information for 
certain purposes. For example, the privacy policy of a medical provider states, "We may disclose 
your [protected health information] for public health activities and purposes [. . .]" |Was03] . Such 
warnings do not constrain the behavior of the policy provider. 

Other policies that prohibit using certain information for a purpose do constrain the behavior 
of the policy provider. Examples include the privacy policy of Yahoo! Email, which states that 
"Yahoo! 's practice is not to use the content of messages stored in your Yahoo! Mail account for 
marketing purposes" jYahlOb[ emphasis added]. 

Some policies even limit the use of certain information to an explicit list of purposes. The privacy 
policy of The Bank of America states, "Employees are authorized to access Customer Information 
for business purposes only." |Bann5l emphasis added]. The HIPAA Privacy Rule [Offn^j requnes 
that covered entities (e.g., health care providers and business partners) only use or disclose protected 
health information about a patient with that patient's written authorization or: 

[. . .] for the following purposes or situations: (1) To the Individual [...]; (2) Treatment, 
Payment, and Health Care Operations; (3) Opportunity to Agree or Object; (4) Incident 
to an otherwise permitted use and disclosure; (5) Public Interest and Benefit Activities; 
and (6) Limited Data Set for the purposes of research, public health or health care 
operations. 

These examples show that verifying that an organization obeys a privacy policy requires a 
semantics of purpose requirements. In particular, enforcement requires the ability to determine 
that the organization under scrutiny obeys at least two classes of purpose requirements. As shown 
in the example rule from Yahoo!, the first requirement is that the organization does not use certain 
sensitive information for a given purpose. The second, as the example rule from HIPAA shows, is 
that the organization uses certain sensitive information only for a given list of purposes. We call 
the first class of requirements prohibitive (not-for) and the second class restrictive (only- for). Each 
class requires determining whether the organization's behavior is for a purpose or not, but they 
differ in whether this indicates a violation or compliance, respectively. 

For example, consider a physician accessing a medical record. Under the HIPAA Privacy Rule, 
the physician may access the record only for certain purposes such as treatment, research, and 
billing. Thus, for an auditor (either internal or external) to determine whether the physician has 
obeyed the Privacy Rule requires the auditor to determine the purposes for which the physician 
accessed the record. The auditor's ability to determine the purposes behind actions is limited since 
the auditor can only observe the behavior of the physician. As a physician may perform the exact 
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same actions for different purposes, the auditor can never be sure of the purposes behind an action. 
However, if the auditor determines that the record access could not have possibly been for any of 
the purposes allowed under the Privacy Rule, then the auditor knows that the physician violated 
the policy. 

Manual enforcement of these privacy policies is labor intensive and error prone. Thus, to 
reduce costs and make their operations more trustworthy, organizations would like to automate 
the enforcement of the privacy policies governing their operations; tool support for this activity is 
beginning to emerge in the market. For example, Fair Warning offers automated services for the 
detection of privacy breaches in a hospital setting |Faij . Meanwhile, previous research has purposed 
formal methods to enforce purpose requirements |AKSXn2[ IBBLn5[ IHAOSi IAFHtI IbIM IPGYOSi 
l,TSNSn9l iNBL+ini IEKWBTT] . 

However, each of these endeavors start by assuming that actions or sequences of actions are 
labeled with the purposes they are for. They avoid analyzing the meaning of purpose and provide 
no method of performing this labeling other than through intuition alone. The absence of a formal 
semantics to guide this determination has hampered the development of methods for ensuring 
policy compliance. Such a definition would provide insights into how to develop tools that identify 
suspicious accesses in need of detailed auditing and algorithms for determining which purposes an 
action could possibly be for. Such a definition would also show which enforcement approaches are 
most accurate. More fundamentally, such a definition could frame the scientific basis of a societal 
and legal understanding of purpose and of privacy policies that use the notion of purpose. Such 
a foundation can, for example, guide implementers as they codify in software an organization's 
interpretation of internal and government-imposed privacy policies. 

1.1 Solution Approach 

The goal of this work is to study the meaning of purpose in the context of enforcing privacy policies 
and propose formal definitions suitable for automating the enforcement of purpose requirements. 
Since post-hoc auditing provides the perspective often required to determine the purpose of an 
action, we focus on automated auditing. However, we believe our semantics is applicable to other 
formal methods and may also clarify informal reasoning. 

We find that planning is central to the meaning of purpose. We see the role of planning in the 
definition of the sense of the word "purpose" most relevant to our work |OED89j : 

The object for which anything is done or made, or for which it exists; the result or effect 
intended or sought; end, aim. 

Similarly, work on cognitive psychology calls purpose "the central determinant of behavior" |DKP96t 
pl9]. If our auditors are concerned with rational auditees (the person or organization being au- 
dited), then we may assume the auditee uses a plan to determine what actions it will perform in 
its attempt to achieve its purposes. We (as have philosophers |Tay66| ) conclude that if an auditee 
selects to perform an action a while planning to achieve the purpose p, then the auditee's action a 
is for the purpose p. In this paper, we make these notions formal. 

1.2 Overview of Contributions 

We first present an example that illustrates key factors in determining whether an action is for a 
purpose or not. We find that the auditor should model the auditee as an agent that interacts with 
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an environment model. The environment model shows how the actions the auditee can perform 
affect the state of the environment. It also models how well each state satisfies each purpose that 
the modeled auditee might possibly find motivating. Limiting consideration to one purpose, the 
environment model becomes a Markov Decision Process (MDP) where the degree of satisfaction of 
that purpose is the reward function of the MDP. If the auditee is motivated to act by only that 
purpose, then the auditee's actions must correspond to an optimal plan for this MDP and these 
actions are for that purpose. Additionally, we use a stricter definition of optimal than standard 
MDPs to reject redundant actions that neither decrease nor increase the total reward. We formalize 
this model in Section [3l 

For example, consider a physician ordering a medical test and an auditor attempting to de- 
termine whether the physician could have ordered this test for the purpose of treatment (and is 
therefore in compliance with the HIPAA Privacy Rule). The auditor would examine an MDP 
modeling the physician's environment with the quality of treatment as the reward function to be 
optimized. If no optimal plans for this MDP involve ordering the test, then the auditor can conclude 
definitively that the physician did not order the test for treatment. 

We make this auditing process formal in Section H] where we discuss the ramifications of the 
auditor only observing the behaviors of the auditee and not the underlying planning process of the 
auditee that resulted in these behaviors. We show that in some circumstances, the auditor can still 
acquire enough information to determine that the auditee violated the privacy policy. To do so, 
the auditor must first use our MDP model to construct all the possible behaviors that the privacy 
policy allows and then compare it with all the behaviors of the auditee that could have resulted 
in the observed auditing log. Section [5] presents an algorithm for auditing based on our formal 
definitions, illustrating the relevance of our work. 

The semantics discussed thus far is sufficient to put the previous work on enforcing privacy 
policies on firm semantic ground. In Section [6l we do so and discuss the strengths and weaknesses 
of each such approach. In particular, we find that each approach may be viewed as a method of 
enforcing the policy given the set of all possible allowed behaviors, an intermediate result of our 
analysis. We compare the previous auditing approaches, which differ in their trade-offs between 
auditing complexity and accuracy of representing this set of behaviors. 

Most auditees are actually interested in multiple purposes and select plans that simultaneously 
satisfy as many of the desired purposes as possible. Handling the interactions between purposes 
complicates our semantics. In particular, actions selected by a single plan may be for different 
purposes. In Section [TJ we present examples showing when our semantics can extend to handle 
multiple purposes and when difficulties arise in determining which purposes an action is for when 
an auditee is attempting to satisfy various purposes at once. Currently, the state-of-the-art in the 
understanding of human planning limits our abilities to improve upon our semantics. However, 
as this understanding improves, one may replace our MDP-like formalism with more detailed ones 
while retaining our general framework of defining purpose in terms of planning. 

We end by discussing other related work, future work, and conclusions. Our contributions 
include: 

• The first semantic formalism of when a sequence of actions is for a purpose, 

• An auditing algorithm for this formalism, 

• The resituating of previous policy enforcement methods in our formalism and a comparative 
study of their expressiveness, and 
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• The first attempt to formally consider the effects on auditing caused by interactions among 
multiple purposes. 

Although motivated by our goal to formalize the notions of use and purpose prevalently found 
in privacy policies, our work is more generally applicable to a broad range of policies, such as fiscal 
policies governing travel reimbursement. 

2 Motivation of Our Approach 

We start with an informal example that suggests that an action is for a purpose if the action 
is part of a plan for achieving that purpose. Consider a physician working at a hospital who, as 
a specialist, also owns a private practice that tests for bone damage using a novel technique for 
extracting information from X-ray images. After seeing a patient and taking an X-ray, the physician 
forwards the patient's medical record including the X-ray to his private practice to apply this new 
technology. As this action entails the transmission of protected health information, the physician 
will have violated HIPAA if this transmission is not for one of the purposes HIPAA allows. The 
physician would also run afoul of the hospital's own policies governing when outside consultations 
are permissible unless this action was for a legitimate purpose. Finally, the patient's insurance will 
only reimburse the costs associated with this consultation if a medical reason (purpose) exists for 
them. The physician claims that this consultation was for reaching a diagnosis. As such, it is for 
the purpose of treatment and, therefore, allowed under each of these policies. The hospital auditor, 
however, has selected this action for investigation since the physician's making a referral to his own 
private practice makes the alternate motivation of profit possible. 

Whether or not the physician violated these policies depends upon details not presented in the 
above description. For example, we would expect the auditor to ask questions such as: (1) Was 
the test relevant to the patient's condition? (2) Did the patient benefit medically from having the 
test? (3) Was this the best option for the patient? We will introduce these details as we introduce 
each of the factors relevant to the purposes behind the physician's actions. 

States and Actions. Sometimes the purposes for which an action is taken depend upon the 
previous actions and the state of the system. In the above example, whether or not the test is 
relevant depends upon the condition of the patient, that is, the state that the patient is in. 

While an auditor could model the act of transmitting the record as two (or more) different 
actions based upon the state of the patient, modeling two concepts with one formalism could 
introduce errors. A better approach is to model the state of the system. The state captures the 
context in which the physician takes an action and allows for the purposes of an action to depend 
upon the actions that precede it. 

The physician's own actions also affect the state of the system and, thus, the purposes for which 
his actions are. For example, had the physician transmitted the patient's medical record before 
taking the X-ray, then the transmission could not have been for treatment since the physician's 
private practice only operates on X-rays and would have no use for the record without the X-ray. 

The above example illustrates that when an action is for a purpose, the action is part of a 
sequence of actions that can lead to a state in which some goal associated with the purpose is 
achieved. In the example, the goal is reaching a diagnosis. Only when the X-ray is first added to 
the record is this goal reached. 
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Non-redundancy. Some actions, however, may be part of such a sequence without actually being 
for the purpose. For example, suppose that the patient's X-ray clearly shows the patient's problem. 
Then, the physician can reach a diagnosis without sending the record to the private practice. Thus, 
while both taking the X-ray and sending the medical record might be part of a sequence of actions 
that leads to achieving a diagnosis, the transmission does not actually contribute to achieving the 
diagnosis: the physician could omit it and the diagnosis could still be reached. 

Prom this example, it may be tempting to conclude that an action is for a purpose only if 
that action is necessary to achieve that purpose. However, consider a physician who has a choice 
between two specialists to whom to send the medical record and must do so to reach a diagnosis. 
In this scenario, the physician's sending the record to the first specialist is not necessary since he 
could send it to the second. Likewise, sending the record to the second specialist is not necessary. 
Yet, the physician must send the record to one or the other specialist and that transmission will 
be for the purpose of diagnosis. Thus, an action may be for a purpose without being necessary for 
achieving the purpose. 

Rather than necessity, we use the weaker notion of non-redundancy found in work on the 
semantics of causation (e.g., |Mac74j ) . Given a sequence of actions that achieves a goal, an action 
in it is redundant if that sequence with that action removed (and otherwise unchanged) also achieves 
the goal. An action is non-redundant if removing that action from the sequence would result in 
the goal no longer being achieved. Thus, non-redundancy may be viewed as necessity under an 
otherwise fixed sequence of actions. 

For example, suppose the physician decides to send the medical record to the first specialist. 
Then, the sequence of actions modified by removing this action would not lead to a state in which 
a diagnosis is reached. Thus, the transmission of the medical record to the first specialist is non- 
redundant. However, had the X-ray revealed to the physician the diagnosis without needing to 
send it to a specialist, the sequence of actions that results from removing the transmission from 
the original sequence would still result in a diagnosis. Thus, the transmission would be redundant. 

Quantitative Purposes. Above we implicitly presumed that the diagnosis from each specialist 
had equal quality. This need not be the case. Indeed, many purposes are actually fulfilled to 
varying degrees. For example, the purpose of marketing is never completely achieved since there is 
always more marketing to do. Thus, we model a purpose by assigning to each state-action pair a 
number that describes how well that action fulfills that purpose when performed in that state. We 
require that the physician selects the test that maximizes the quality of the diagnosis as determined 
by total purpose score accumulated over all his actions. 

Probabilistic Systems. The success of many medical tests and procedures is probabilistic. For 
example, with some probability the physician's test may fail to reach a diagnosis. The physician 
would still have transmitted the medical record for the purpose of diagnosis even if the test failed 
to reach one. This possibility affects our semantics of purpose: now an action may be for a purpose 
even if that purpose is never achieved. 

To account for such probabilistic events, we model the environment in which the physician 
operates as probabilistic. For an action to be for a purpose, we require that there be a non-zero 
probability of the purpose being achieved and that the physician attempts to maximize the expected 
reward. In essence, we require that the physician attempts to achieve a diagnosis. Thus, the 
auditee's plan determines the purposes behind his actions rather than just the actions themselves. 
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3 Planning for a Purpose 



In this section, we present a formalism for planning that accounts for quantitative purposes, prob- 
abilistic systems and non-redundancy. We start by modeling the environment in which the auditee 
operates as a Markov Decision Process (MDP) — a natural model for probabilistic systems. The 
reward function of the MDP quantifies the degree of satisfaction of a purpose upon taking an ac- 
tion from a state. If the auditee is motivated to action by only that purpose, then the auditee's 
actions must correspond to an optimal plan for this MDP and these actions are for that purpose. 
We develop a stricter definition of optimal than standard MDPs, which we call NMDPs for Non- 
redundant MDP, to reject redundant actions that neither decrease nor increase the total reward. 
We end with an example illustrating the use of an NMDP to model an audited environment. 

3.1 Markov Decision Processes 

An MDP may be thought of as a probabilistic automaton where transitions are labeled with a 
reward in addition to an action. Rather than having accepting or goal states, the "goal" of a MDP 
is maximizing the total reward over time. 

An MDP is a tuple m = (Q, A, t, r, 7) where Q is a set of states, ^ is a set of actions, t : Q x ^ — > 
P(Q) a transition function from a state and an action to a distribution over states (represented 
asX'(Q)), r : Qx^— T-Ra reward function, and 7 a discount factor such that < 7 < 1. For 
each state q in Q, the agent using the MDP to plan selects an action a from A to perform. Upon 
performing the action a in the state q, the agent receives the reward r{q,a). The environment 
then transitions to a new state q' with probability /u(g') where fi is the distribution provided by 
t{q, a). The goal of the agent is to select actions to maximize its expected total discounted reward 
E [X^^o 7V«] where i G N (the set of natural numbers) ranges over time modeled as discrete steps. 
Pi is the reward at time i, and the expectation is taken over the probabilistic transitions. 

We formalize the agent's plan as a stationary strategy (commonly called a "policy", but we 
reserve that word for privacy policies). A stationary strategy is a function a from the state space 
Q to the set A of actions (i.e., a : Q ^ A) such that at a state q in Q, the agent always selects to 
perform the action cr{q). Given a strategy a for an MDP m, its expected total discounted reward 
is 

Vm{a, q) = r{q, a{q)) + 7 t{q, a{q)){q') * Vm{(T, q') 

The agent selects one of the strategies that optimizes this equation. We denote this set of optimal 
strategies as opt((Q, r, 7)), or when the transition system is clear from context, as opt(r). 
Such strategies are sufficient to maximize the agent's expected total discounted reward despite 
only depending upon the MDP's current state. 

Given the strategy a and the actual results of the probabilistic transitions yielded by t, the agent 
exhibits an execution. We represent this execution as an infinite sequence e = [gi, oi, 52, a2> ■ ■ ■] of 
alternating states and actions starting with a state, where qi is the ith state that the agent was in 
and is the ith action the agent took, for all i in N. We say an execution e is consistent with a 
strategy u iff = (T{qi) for all i in N where Oj is the ith action in e and qi is the ith state in e. We 
call a finite prefix of an execution a behavior. A behavior is consistent with a strategy if it can be 
extended to an execution consistent with that strategy. 

Under this formalism, the auditee plays the role of the agent optimizing the MDP to plan. We 
presume that each purpose may be modeled as a reward function. That is, we assume the degree 
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to which a purpose is satisfied may be captured by a function from states and actions to a real 
number. The higher the number, the higher the degree to which that purpose is satisfied. When the 
auditee wants to plan for a purpose p, it uses a reward function, r^, such that r^((/, a) is the degree 
to which taking the action a from state q aids the purpose p. We also assume that the expected 
total discounted reward can capture the degree to which a purpose is satisfied over time. We say 
that the auditee plans for the purpose p when the auditee adopts a strategy a that is optimal for 
the MDP {Q, A,t,r'P ,j) . The appendix provides additional background information on MDPs. 

3.2 Non-redundancy 

MDPs do not require that strategies be non-redundant. Even given that the auditee had an 
execution e from using a strategy a in opt(rP), some actions in e might not be for the purpose p. 
The reason is that some actions may be redundant despite being costless. The MDP optimization 
criterion behind opt prevents redundant actions from delaying the achievement of a goal as the 
reward associated with that goal would be further discounted making such redundant actions 
sub-optimal. However, the optimization criterion is not affected by redundant actions when they 
appear after all actions that provide non-zero rewards. Intuitively, the hypothetical agent planning 
only for the purpose in question would not perform such unneeded actions even if they have zero 
reward. Thus, to create our formalism of non-redundant MDPs (NMDPs), we replace opt with a 
new optimization criterion opt* that prevents these redundant actions while maintaining the same 
transition structure as a standard MDP. 

To account for redundant actions, we must first contrast that with doing nothing. Thus, we 
introduce a distinguished action N that stands for doing nothing. For all states q, N labels a 
transition with zero reward (i.e., r(g, N) = 0) that is a self-loop (i.e., t{q,N){q) = 1). (We could 
put N on only the subset of states that represent possible stopping points by slightly complicating 
our formalism.) Since we only allow deterministic stationary strategies and N only labels self- loops, 
this decision is irrevocable: once nothing is done, it is done forever. As selecting to do nothing 
results in only zero rewards henceforth, it may be viewed as stopping with the previously acquired 
total discounted reward. 

Given an execution e, let active(e) denote the prefix of e before the first instance of the nothing 
actions, active(e) will be equal to e in the case where e does not contain the nothing action. 

We use the idea of nothing to make formal when one execution intuitively contain more actions 
than another despite both being of infinite length. An execution ei is a proper sub-execution of an 
execution 62 if and only if active(ei) is a proper subsequence of active(e2) using the standard notion 
of subsequence. Note if ei does not contain the nothing action, it cannot be a proper sub-execution 
of any execution. 

To compare strategies, we construct all the executions they could produce. To do so, let a 
contingency k be a function from Q x ^ x N to Q such that K{q, a, i) is the state that results from 
taking the action a in the state q the rth time. We say that a contingency k is consistent with 
an MDP iff k only picks states to which the transition function t of the MDP assigns a non-zero 
probability to (i.e., for all q in Q, a in A, and i in N, t(q,a){K{q,a,i)) > 0). Given an MDP m, 
let m{q, k) be the possibly infinite state model that results of having k resolve all the probabilistic 
choices in m and having the model start in state q. Let m{q, k, a) denote the execution that results 
from using the strategy a and state q in the non-probabilistic model m{q, k). Henceforth, we only 
consider contingencies consistent with the model under discussion. 

Given two strategies a and a', we write a' ~< a and only if for all contingencies k and states 
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Figure 1: The environment model mgx that the physician used. Circles represent states, block 
arrows denote possible actions, and squiggly arrows denote probabilistic outcomes. Self-loops of 
zero reward under all actions, including the special action N, are not shown. 

m{q,K,,a') is a proper sub-execution of or equal to m{q, K,a), and for at least one contingency 
k' and state q', m{q' , k' ,a') is a proper sub-execution m{q',K',a). Intuitively, a' proves that a 
produces a redundant execution under k' and q' . We define opt*(r) to be the subset of opt(r) 
holding only strategies a such that for no a' € opt(r) does a' -< a. The following theorem, proved 
in the appendix, shows that non-redundant optimal strategies always exist. 

Theorem 1. For all environment models m, opt*(m) is not empty. 
3.3 Example 

Suppose an auditor is inspecting a hospital and comes across a physician referring a medical record 
to his own private practice for analysis of an X-ray as described in Section [2j As physicians may 
only make such referrals for the purpose of treatment (treat), the auditor may find the physi- 
cian's behavior suspicious. To investigate, the auditor may formally model the hospital using our 
formalism. 

The auditor would construct the NMDP = (Qexj ^ex, ^exi ''ex^^*) 7ex) shown in Figured) The 
figure conveys all components of the NMDP except 7ex- For instance, the block arrow from the 
state 1 labeled take and the squiggly arrows leaving it denote that after the agent performs the 
action take from state 1, the environment will transition to the state 2 with probability 0.9 and to 
state 4 with probability of 0.1 (i.e., tex(l, take)(2) = 0.9 and tex(l5 take)(4) = 0.1). The number over 
the block arrow further indicates the degree to which the action satisfies the purpose of treat. In 
this instance, it shows that rg5|f^*(l, take) = 0. This transition models the physician taking an X-ray. 
With probability 0.9, he is able to make a diagnosis right away (from state 2); with probability 0.1, 
he must send the X-ray to his practice to make a diagnosis. Similarly, the transition from state 4 
models that his practice's test has a 0.8 success rate of making a diagnosis; with probability 0.2, 
no diagnosis is ever reached. 

Using the model, the auditor computes opt(rg5|f^*), which consists of those strategies that max- 
imizes the expected total discounted degree of satisfaction of the purpose of treatment where the 
expectation is over the probabilistic transitions of the model. opt(rg5f^*) includes the appropriate 
strategy ui where (Ti(1) = take, <7i(4) = send, (Ti(2) = cri(3) = (Ti(5) = diagnose, and cti(6) = N. 
Furthermore, opt(rg5|f^*) excludes the redundant strategy (72 that performs a redundant send where 
(72 is the same as ai except for o"2(2) = send. Performing the extra action send delays the reward 
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of 12 for achieving a diagnosis resulting in its discounted reward being * 12 instead of 7ex * 12 
and, thus, the strategy is not optimal. 

However, opt{r^^^^) does include the redundant strategy (T3 that is the same as ai except for 
(73(6) = send. opt(rg^^^*) includes this strategy despite the send actions from state 6 being redundant 
since no positive rewards follow the send actions. Fortunately, opt*(rg^^^*) does not include (T3 since 
(Ji is both in opt(rgJf^*) and ai ^ a^. To see that ai -< (T3 note that for every contingency k and 
state g, the mex(g, n, ai) has the form b followed by an finite sequence of nothing actions (interleaved 
with the state 6) for some finite prefix b. For the same k, mex(Q, k, a^) has the form b followed by an 
infinite sequence of send actions (interleaved with the state 6) for the same b. Thus, rriexiq, K,ai) 
is a proper sub-execution of mex{q, k, 0-3). 

4 Auditing 

In the above example, the auditor constructed a model of the environment in which the auditee 
operates. The auditor must use the model to determine if the auditee obeyed the policy. We 
first discuss this process for auditing restrictive policy rules and revisit the above example. Then, 
we discuss the process for prohibitive policy rules. In the next section, we provide an auditing 
algorithm that automates comparing the auditee's behavior, as recorded in a log, to the set of 
allowed behaviors. 

4.1 Auditing Restrictive Rules 

Suppose that an auditor would like to determine whether an auditee performed some logged actions 
only for the purpose p. The auditor can compare the logged behavior to the behavior that a 
hypothetical agent would perform when planning for the purpose p. In particular, the hypothetical 
agent selects a strategy from opt*((Q, ^, t, r^, 7)) where Q, A, and t models the environment of 
the auditee; is a reward function modeling the degree to which the purpose p is satisfied; and 7 
is an appropriately selected discounting factor. If the logged behavior of the auditee would never 
have been performed by the hypothetical agent, then the auditor knows that the auditee violated 
the policy. 

In particular, the auditor must consider all the possible behaviors the hypothetical agent could 
have performed. For a model m, let behv*(rP) represent this set where a finite prefix b of an 
execution is in behv*(rP) if and only if there exists a strategy a in opt*(r*'), a contingency k, and 
a state q such that 6 is a subsequence of m{q, k, a). 

The auditor must compare behv*(r^') to the set of all behaviors that could have caused the 
auditor to observe the log that he did. We presume that the log i was created by a process log that 
records features of the current behavior. That is, log: i? — > L where B is the set of behaviors and 
L the set of logs, and £ = log(6) where b is the prefix of the actual execution of the environment 
available at the time of auditing. The auditor must consider all the behaviors in \og~^{£) as 
possible where log~^ is the inverse of the logging function. In the best case for the auditor, the log 
records the whole prefix b of the execution that transpired until the time of auditing, in which case 
log-^^) = {£}. 

If log ^{£) n behv*(rP) is empty, then the auditor may conclude that the auditee did not plan 
for the purpose p, and, thus, violated the rule that auditee must only perform the actions recorded 
in i for the purpose p; otherwise, the auditor must consider it possible that the auditee planned for 
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the purpose p. 

If log~^(^) C behv*(r^), the auditor might be tempted to conclude that the auditee surely obeyed 
the policy rule. However, as illustrated in the second example below, this is not necessarily true. 
The problem is that \og~^{£) might have a non-empty intersection with behv*(r^ ) for some other 
purpose p'. In this case, the auditee might have been actually planning for the purpose p' instead of 
p. Indeed, given the likelihood of such other purposes for non-trivial scenarios, we consider proving 
compliance practically impossible. However, this incapability is of little consequence: \og~^{£) C 
behv*(r^) does imply that the auditee is behaving as though he is obeying the policy. That is, in 
the worse case, the auditee is still doing the right things even if for the wrong reasons. 

4.2 Example 

Below we revisit the example of Section [3. 31 We consider two cases. In the first, the auditor shows 
that the physician violated the policy. In the second, auditing is inconclusive. 

Violation Found. Suppose after constructing the model as above in Section 13.31 the auditor 
maps the actions recorded in the access log ii to the actions of the model rriex, and finds \og~^{ii) 
holds only a single behavior: bi = [1, take, 2, send, 3, diagnose, 6, N, 6]. Next, using opt*{r^^^^), as 
computed above, the auditor constructs the set behv*(rg5^^^*) of all behaviors an agent planning for 
treatment might exhibit. The auditor would find that bi is not in behv*(rg5f^*). 

To see this, note that every execution ei that has bi as a prefix is generated from a strategy 
a such that cr(2) = send. The strategy a2 from Section 13.31 is one such strategy. None of these 
strategies are members of opt{r^^^^) for the same reason as a2 is not a member. Thus, bi cannot be 
in behv*(rg5|f^*). As \og~^{£) D hehv* (r^^^^) is empty, the audit reveals that the physician violated 
the policy. 

Inconclusive. Now suppose that the auditor sees a different log £2 such that log" ^(£2) = {^2} 
where 62 = [Ij take, 4, send, 5, diagnose, 6, N, 6]. In this case, our formalism would not find a violation 
since 62 is in behv*(rgx^^*). In particular, the strategy ai from above produces the behavior 62 under 
the contingency that selects the bottom probabilistic transition from state 1 to state 4 under the 
action take. 

Nevertheless, the auditor cannot be sure that the physician obeyed the policy. For example, 
consider the NMDP m'^^ that is mgx altered to use the reward function rex°^'* instead of rl'^^^ . rex°''* 
assigns a reward of zero to all transitions except for the send actions from states 2 and 4, to which 
it assigns a reward of 9. cJi is in opt*(rex°^'*) meaning that not only the same actions (those in 
62), but even the exact same strategy can be either for the allowed purpose treat or the disallowed 
purpose profit. Thus, if the physician did refer the record to his practice for profit, he cannot be 
caught as he has tenable deniability of his ulterior motive of profit. 

4.3 Auditing Prohibitive Rules 

In the above example, the auditor was enforcing the rule that the physician's actions be only for 
treatment. Now, consider auditing to enforce the rule the that physician's actions are not for 
personal profit. After seeing the log i, the auditor could check whether \og~^{i) n behv*(rex°^'*) is 
empty. If so, then the auditor knows that the policy was obeyed. If not, then the auditor cannot 
prove nor disprove a violation. In the above example, just as the auditor is unsure whether the 
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actions were for the required purpose of treatment, the auditor is unsure whether the actions are 
not for the prohibited purpose of profit. 

An auditor might decide to investigate some of the cases where \og~^{£) fl behv*(rex°^'*) is not 
empty. In this case, the auditor could limit his attention to only those possible violations of a 
prohibitive rule that cannot be explained away by some allowed purpose. For example, in the 
inconclusive example above, the physician's actions can be explained with the allowed purpose of 
treatment. As the physician has tenable deniability, it is unlikely that investigating his actions 
would be a productive use of the auditor's time. Thus, the auditor should limit his attention to 
those logs £ such that both \og~^{£) n behv*(rex°''*) is non-empty and log~^(^) n behv*(rg!^^^*) is 
empty. 

A similar additional check using disallowed purposes could be applied to enforcing restrictive 
rules. However, for restrictive rules, this check would identify cases where the auditee's behavior 
could have been either for the allowed purpose or a disallowed purpose. Thus, it would serve 
to find additional cases to investigate and increase the auditor's workload rather than reduce it. 
Furthermore, the auditee would have tenable deniability for these possible ulterior motives, making 
these investigations a poor use of the auditor's time. 

5 Auditing Algorithm 

We would like to automate the auditing process described above. To this end, we present in FigureE] 
an algorithm AuDiT that aids the auditor in comparing the log to the set of allowed behaviors. As 
we are not interested in the details of the logging process and would like to focus on the planning 
aspects of our semantics, we limit our attention to the case where \og(b) = b. As proved below 
(Theorem [2]) , AuDiT(m,6) returns true if and only if \og~^(b) n behv*(m) is empty. In the case of 
a restrictive rule, the auditor may conclude that the policy was violated when Audit returns true. 
In case of a prohibitive rule, the auditor may conclude the policy was obeyed when Audit returns 
true. 

Audit operates in two steps. The first checks to make sure that the behavior b is not inherently 
redundant (lines 01-05). If it is, then log~^(6) n behv*(m) will be empty and the algorithm returns 
true. Audit checks b by comparing the actions taken in each state to doing nothing. If the expected 
total discounted reward for doing nothing in a state q is higher than that for doing the action a in 
q, then a introduces redundancy into any strategy a such that a{q) = a. Thus, if 6 = [. . . , q, a, . . .], 
we may conclude that log" ""^(6) D behv*(m) is empty. 

The second step compares the optimal values of two MDPs. One of the them is the NMDP m 
treated as an MDP, which is already optimized during the first step. The other m' is constructed 
from m (lines 07-17) so that only the actions in the log b are selected during optimization. If the 
expected total discounted reward of each of these MDPs is unequal, then log~^(6) n behv*(m) is 
empty. 

Below we formalize these ideas. Lemma [T] justifies our two step approach while Lemmas [2] 
and [3] justify how we perform the first and second step, respectively. They allow us to conclude 
the correctness of our algorithm in Theorem [2l We defer proofs and additional propositions to the 
appendix. 
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AuDIT((Q,^,t,r, 7), [qo,ai,qi,...an,qn])- 

01 V^:= so\veMDP{{Q, A,t,r,-f)) 

02 for(i := 0; i < n; 

03 if(ai+i / N): 

04 mQ^][a^+l]+lE\%tMa^+l][j]*V^[j] < 0): 

05 return true 

06 r* := 

07 for(i :=0;i < \Q\; 

08 for(/c ■.= 0;k< \A\; k++): 

09 r'[j][k] := r[j][k] 

10 if(r* < absoluteValue(r[j][/i;]): 

11 r* := absoluteValue(r[j][A;]) 

12 a;:=2*rV(l-7) + l 

13 for(i := 0; i < n; 

14 for(A; := 0; A; < |^|; k++): 

15 if(A; ^ flj+i): 

16 r'[qi][k] := -co 

17 m' := {Q,A,t,r',-f) 

18 V;;, :=solveMDP((Q,^,t,r',7)) 

19 for(i :=0;i < \Q\; 

20 iTCb1=K:,b1): 

21 return false 

22 return true 

Figure 2: The algorithm Audit. solveMDP may be any MDP solving algorithm. The algorithm 
assumes functions are represented as arrays and states and actions are represented as indexes into 
these arrays. 
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5.1 Useless States and the Two Steps 

We say an action is useless at a state if taking it would always lead to redundancy. Formally, let 
the set Um be the subset oi Qx A such that {q, a) is in Um if and only if a 7^ N and for all strategies 
cr, Qm{(^, q,a) <0 where Qm(<7, q, a) = r{q, a) + 7 Yjq' * Kn(o", q'). 

We call ((7, a) in set useless since any strategy a such that a{q) = a could be replaced by 
a strategy a' that is the same as a except for having (T'{q) = N without lowering the expected 
total discounted reward. To make this formal, let U{a) be a strategy such that U{a){q) = N if 
{q,a{q)) G U and U{a){q) = a{q) otherwise. The following justifies calling these pairs useless: for 
all a and q, Vm{cr,q) < Fm(C/m(c), (Proposition [1]) . 

We are also interested in the set strg(6) of strategies that could have resulted in the behavior b: 
strg(6) = {a e Q ^ A l^i < n.Oj+i = a{qi) } where b = [qo,ai,qi,a2, ■ ■ ■ ,an,gn]- 

Lemma 1. For all environment models m and all behaviors b = [qo, ai, qi, . . . , On, qn 

], \og-\b) n 

behv*(m) is empty if and only if (1) there exists i such that < i < n and (gj,aj+i) € Um or (2) 
strg(6) n opt(m) is empty, 

Thus, checking whether log~^(6) n behv*(m) is empty has been reduced to checking the two 
conditions (1) and (2). We explain how to check each of these in the next two sections. 

5.2 Step 1: Inherent Redundancy 

Rather than construct Um explicitly, we use the following lemma to check condition (1). The lemma 
uses the definition Qm{q, a) = r{q, a) + 7 Ylq' o){q') * VmW) where Vm{q) = max^- Vm{(T, a)- 

Lemma 2. For all environment models m, states q, and actions a, {q, a) is in Um if and only if 
a 7^ N and Qm{q, a) < 0. 

5.3 Step 2: Checking Optimahty 

To check (2), we construct a model m' from m that limits the optimization to selecting a strategy 
that can cause the observed behavior b. To do so, we adjust the reward function of m so that the ac- 
tions taken in b are always taken by the optimal strategies of m'. That is, if 6 = [q(),ai,qi, . . . , an,qn], 
then for each qi and Oj+i, we replace the reward for taking an action a' other than Oj+i from the 
state q with a negative reward —u that is so low as to assure that the action a' would not be used 
by any optimal strategy. We use lo > 2r*/(l — 7) where r* is the reward with the largest magnitude 
appearing in m since the total discounted reward is bounded from below by — r*/ (1 — 7) and from 
above by r*/{l - 7) (recall that ESo7*^* = - 7))- 

We formally define m' to be fix(m, b) where fix(m, []) = m and 

fix((Q,^,t,r,7), [qo,ai,qi, . . . ,a„,g„]) = fix((Q, ^, t, r', 7), [qi,.. .,an,qn\) 

where r'{qQ,a) = -co for all a ^ ai and r'{qQ,ai) = r{qo,ai). The construction fix has the 
following useful property: strg(6) n opt(m) is empty if and only if opt(fix(m, b)) n opt(m) is empty 
(Proposition 111 p . This property is useful since testing whether opt(m) nopt(fix(?7i, b)) is empty may 
be reduced to simply comparing their optimal values: opt(m) nopt(fix(m, b)) is empty if and only if 
for all states q, max^ Vfjx(m,6) (o'j q) / niax^ Vm{cr, q) (Proposition [T2]) . Fortunately, algorithms exist 
for finding the optimal value of MDPs (see, e.g., |RN03j ). 

These two propositions combine to yield the next lemma, which justifies how we conduct testing 
for the second condition of Lemma [1] in the second step of Audit. 
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Lemma 3. For all environment models m and behaviors h, strg(6) n opt(?7i) is empty if and only 
if for all q, max^ ^fix(m,b) (o", q) / max^ Vmicr, q). 

These lemmas combine with reasoning about the actual code of the program to yield its cor- 
rectness. 

Theorem 2. For all environment models m and behaviors b, AUDlT(m, b) returns true if and only 
if\og^^{b) n behv*(m) is empty. 

The running time of the algorithm is dominated by the two MDP optimizations. These may be 
done exactly by reducing the optimization to a system of linear equations |d'E63j . Such systems 
may be solved in polynomial time [Kha791 IKar84j . However, in practice, large systems are often 
difficult to solve. Fortunately, a large number of algorithms for making iterative approximations 
exist whose run time depends on the quality of the approximation. (See |LDK95] for a discussion.) 

6 Applying our Formalism to Past Methods 

Past methods of enforcing purpose requirements have not provided methods of assigning purposes 
to sequences of actions. Rather, they presume that the auditor (or someone else) already has a 
method of determining which behaviors are for a purpose. In essence, these methods presuppose 
that the auditor already has the set of allowed behaviors behv*(rP) for the purpose p that he is en- 
forcing. These methods differ in their intensional representations of the set behv*(rP). Thus, some 
may represent a given set exactly while others may only be able to approximate it. These differ- 
ences mainly arise from the different mechanisms they use to ensure that the auditee only exhibits 
behaviors from behv*(r^). We use our semantics to study how reasonable these approximations 
are. 

Byun et al. use role-based access control |San96j to consider purposes |BBLn5l [BLMI INBL+ 10] . 
They associate purposes with sensitive resources and with roles, and their method only grants the 
user access to the resource when the purpose of the user's role matches the resource's purpose. 
The method does not, however, explain how to determine which purposes to associate with which 
roles. Furthermore, a user in a role can perform actions that do not fit the purposes associated with 
his role allowing him to use the resource for a purpose other than the intended one. Thus, their 
method is only capable of enforcing policies when there exists some subset A of the set of actions 
A such that behv*(rP) is equal to the set of all interleavings of A with Q of finite but unbounded 
length (i.e., behv*(rP) = (Q x A)*:Q where : is append raised to work over sets in the standard 
pairwise manner). The subset A corresponds to those actions that use a resource with the same 
purpose as the auditee's role. Despite these limitations, their method can implement the run-time 
enforcement used at some organizations, such as a hospital that allows physicians access to any 
record to avoid denying access in time-critical emergencies. However, it does not allow for the fine- 
grain distinctions used during post-hoc auditing done at some hospitals to ensure that physicians 
do not abuse their privileges. 

Al-Fedaghi uses the work of Byun et al. as a starting point but concludes that rather than as- 
sociating purposes with roles, one should associate purposes with sequences of actions |AF07j . 
Influenced by Al-Fedaghi, Jafari et al. adopt a similar position calling these sequences work- 
flows |JSNS09] . The set of workflows allowed for a purpose p corresponds to behv*(rP). They 
do not provide a formal method of determining which workflows belong in the allowed set. They do 
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not consider probabilistic transitions and the intuition they supply suggests that they would only 
include workflows that successfully achieves or improves the purpose. Thus, our approach appears 
more lenient by including some behaviors that fail to improve the purpose. 

Others have adopted a hybrid approach allowing for the roles of an auditee to change based 
on the state of the system |PGY08( IEKWB11| . These changes effectively allow role-based access 
control to simulate the workflow methods to be just as expressive while introducing a level of 
indirection inhabited by dynamic roles. 

Agrawal et al. use a query intrusion model to enforce purpose requirements that operates in 
a manner similar to intrusion detection [AKSX02] . Their method flags a request for access as a 
possible violation if the request claims to be for a purpose despite being dissimilar to previous 
requests for the same purpose. To avoid false positives, the set of allowed behaviors behv*(rP) 
would have to be small or have a pattern that the query intrusion model could recognize. 

Jif is a language extension to Java designed to enforce requirements on the flows of information in 
a program [CMVZdO] . Hayati and Abadi explain how to reduce purpose requirements to information 
flow properties that Jif can enforce |HA05j . Their method requires that inputs are labeled with 
the purposes for which the policy allows the program to use them and that each unit of code be 
labeled with the purposes for which that code operates. If information can flow from an input 
statement labeled with one purpose to code labeled for a different purpose, their method produces 
a compile-time type error. (For simplicity, we ignore their use of sub-typing to model sub-purposes.) 
In essence, their method enforces the rule if information i flows to code c, then i and c must be 
labeled with the same purpose. The interesting case is when the code c uses the information i to 
perform some observable action ac,j, such as producing output. Under our semantics, we treat the 
program as the auditee and view the policy as limiting these actions. By directly labeling code, 
their method does not consider the contexts in which these actions occur. Rather the action ac,i 
is aways either allowed or not based on the purpose labels of c and i. By not considering context, 
their method is subject to the same limitations as the method of Byun et al. with the subset A 
being equal to the set of all actions a^i such that c and i have the same label. However, using 
more advanced type systems (e.g., typestate |SY86j ). they might be able extend their method to 
consider the context in which code is executed and increase the method's expressiveness. 

7 Multiple Purposes 

So far, our formalism allows our hypothetical agent to consider only a single purpose. However, 
auditees may perform an action for more than one purpose. In many cases, the auditor may simply 
ignore any action that is not governed by the privacy policy and not relevant to the plans the 
auditee is employing that uses governed actions. 

In the physician example above, the physician already implicitly considered many other purposes 
before even seeing this current patient. For example, the physician presumably performed many 
actions not mentioned in the model in between taking the X-ray, sending it, and making a diagnosis, 
such as going on a coffee break. As these actions are not governed by the privacy policy and neither 
improves nor degrades the diagnosis even indirectly, the auditor may safely ignore them. Thus, our 
semantics can handle multiple purposes in this limited fashion. 

However, in other cases, the interactions between purposes become important. Below we discuss 
two complementary ways that an auditee can consider multiple purposes that produce interactions. 
In the first, the auditee considers one purpose after another. In the second, the auditee attempts to 
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optimize for multiple purposes simultaneously. We find that our semantics may easily be extended to 
handle the first, but difficulties arise for the second. We end the section by considering what features 
a formalism would need to handle simultaneous consideration of purposes and the challenges they 
raise for auditing. 



7.1 Sequential Consideration 

Yahoo! 's privacy policy states that they will not contact children for the purpose of market- 
ing |YahlOa] . Suppose Yahoo! decides to change the name of games . yahoo . com' to [fun, yahoo . com 



because they believe the new name will be easier to market. They notify users of games . yahoo . com 



including children, of the upcoming change so that they may update their bookmarks. 

In this example, the decision to change names, made for marketing, causes Yahoo! to contact 
children. However, we do not feel this is a violation of Yahoo! 's privacy policy. A decision made 
for marketing altered the expected future of Yahoo! in such a way that customer service would 
suffer if Yahoo! did nothing. Thus, to maintain good customer service, Yahoo! made the decision to 
notify users without further consideration of marketing. Since Yahoo! did not consider the purpose 
of marketing while making this decision, contacting the children was not for marketing despite 
Yahoo! considering the implications of changing the name for marketing while making its decision 
to contact children. 

Bratman describes such planning in his work formalizing intentions [Bra87j . He views it as a 
sequence of planning steps in which the intention to act (e.g., to change the name) at one step may 
affect the plans formed at later steps. In particular, each step of planning starts with a model of the 
environment that is refined by the intentions formed by each of the previous planning steps. The 
step then creates a plan for a purpose that further refines the model with new intentions resulting 
from this plan. Thus, a purpose of a previous step may affect the plan formed in a later step for 
a different purpose by constraining the choices available at the later step of planning. We adopt 
the stance that an action selected at a step is for the purpose optimized at that step but not other 
previous purposes affecting the step. 



7.2 Simultaneous Consideration 

At other times, an auditee might consider more than one purpose in the same step. For example, the 
physician may have to both provide quality treatment and respect the patient's financial concerns. 
In this case, the physician may not be able to simultaneously provide the highest quality care at 
the lowest price. The two competing concerns must be balanced and the result may not maximize 
the satisfaction of either of them. 

The traditional way of modeling the simultaneous optimization of multiple rewards is to combine 
them into a single reward using a weighted average over the rewards. Each reward would be 
weighted by how important it is to the auditee performing the optimization. This amalgamation of 
the various purpose rewards makes it difficult to determine for which purpose various actions are 
selected. 

One possibility is to analyze the situation using counterfactual reasoning (see, e.g., |Mac74j ) . 
For example, given that the auditee performed an action a while optimizing a combination of 
purposes pi and p2 , the auditor could ask if the auditee would have still performed the action a 
even if the auditee had not considered the purpose pi and had only optimized the purpose p2- If 
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Figure 3: Model of a traveler deciding whether to fly or drive. Since every transition is deterministic, 
we represent each as a single arrow. Each is labeled with the action name, the rewards for business 
and the rewards for lecturing in that order. Self-loops of zero reward are not shown including all 
those labeled with the nothing action N. 

not, than the auditor could determine that the action was for pi. However, as the next example 
shows, such reasoning is not sufficient to determine the purposes of the actions. 

To show the generality of purposes, we consider an example involving travel reimbursement. 
Consider a Philadelphian who needs to go to New York City for a business meeting with his employer 
and is invited to give a lecture at a conference in Washington, D.C., with his travel expenses 
reimbursed by the conference. He could drive to either New York or Washington (modeled as the 
actions driveNY and driveDC, respectively). However, due to time constraints he cannot drive to 
both of them. To attend both events, he needs to fly to both (modeled as actions flyNY and flyDC). 
As flying is more expensive, both driving actions receives a higher reward than flying (2 instead of 
1), but flying is better than not going (0). Figure [3] models the traveler's environment. 

Given these constraints, he decides to fly to both only to find auditors at both events scrutinizing 
his decision. For example, an auditor working for the conference could find that his flight to 
Washington was not for the lecture since the traveler would have driven had it not been for work. 
If the conference's policy requires that reimbursed flights are only for the lecture, the auditor might 
deny reimbursement. However, the employer seems even less likely to reimburse the traveler for his 
flight to Washington since the flight is redundant for getting to New York. 

However, under the semantics discussed above, each flight would be for both purposes since only 
when the traveler considers both does he decide to take either flight. While having the conference 
reimburse the traveler for his flight to Washington seems reasonable, the idea that they should also 
reimburse him for his flight to New York appears counterintuitive. 

Our approach of sequential planning also cannot explain this example. To plan sequentially, 
the traveler must consider one of the two events first. If, for example, he considers New York first, 
he will decided to drive to New York and then decline the invitation to Washington. Only by 
considering both events at once, does he decide to fly. 

We believe resolving this conflict requires extending our semantics to consider requirements that 
an action be for a purpose (as opposed to not for or only for). Furthermore, we believe that the 
optimization of combinations of purposes does not accurately model human planning with multiple 
purposes. Intuitively, the traveler selects flyDC not for work but also not only for the conference. 
Rather flyDC seems be for the conference under the constraint that it must not prevent the traveler 
from attending the meeting. In the next section, we consider the possibility of modeling human 
planning more accurately. 
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7.3 Modeling Human Planning 

While MDPs are useful for automated planning, they are not specialized for modeling planning 
by humans, leading to the search for more tailored models [Sim551 IGS02j . Simon proposed to 
model humans as having bounded rationality to account for their limitations and their lack of 
information jSim55j . Work on formalizing bounded rationality has resulted in a variety of planning 
principles ranging from the systematic (e.g., Simon's satisficing) to the heuristic (e.g., |Gig02| ). 
However, "[a] comprehensive, coherent theory of bounded rationality is not available" |Sel021 pl4] 
and there still is "a significant amount of unpredictability in how an animal or a human being will 
undertake to solve a problem" such as planning |DKP961 p40]. 

We view creating semantics more closely tied to human planning interesting future work. How- 
ever, modeling human planning may prove complex enough to justify accepting the imperfections 
of semantics such as ours or even heuristic based approaches for finding violations such as the query 
intrusion model discussed above jAKSX02] . 

Despite these difficulties, one could look for discrepancies between a semantics of purpose re- 
quirements and experimental results on planning. In this manner one could judge how closely a 
semantics approximates human planning in the ways relative to purpose requirements. 

In particular, our semantics appears to hold human auditees to too high of a standard: they are 
unlikely to always be able to pick the optimal strategy for a purpose. When enforcing a restrictive 
rule, this strictness could result in the auditor investigating some auditees who honestly planned 
for the only allowed purpose, but failed to find the optimal policy. While such investigations would 
be false positives, they do have the pleasing side-effect of highlighting areas in which an auditee 
could improve his planning. 

In the case of enforcing prohibitive rules, this strictness could cause the auditor to miss some 
violations that do not optimize the prohibited purpose, but, nevertheless, are for the purpose. The 
additional checks proposed at the end of Section l43l could be useful for detecting these violations: if 
the auditee's actions are not consistent with a strategy that optimizes any of the allowed purposes 
but does improve to some degree the prohibited purpose, the actions may warrant extra scrutiny. 

While our semantics is limited by our understanding of human planning, it still reveals concepts 
crucial to the meaning of purpose. Ideas such as planning and non-redundancy will guide future 
investigations on the topic. 

8 Related Work 

We have already covered the most closely related work in Section [6l Below we discuss work on 
related problems and work on purpose from other fields. 

Minimal Disclosure. The works most similar to ours in approach have been on minimal disclo- 
sure, which requires that the amount of information used in granting a request for access should 
be as little as possible while still achieving the purpose behind the request. Massacci, Mylopou- 
los, and Zannone define minimal disclosure for Hippocratic databases |MMZ06] . Barth, Mitchell, 
Datta, and Sundaram study minimal disclosure in the context of workfiows |BMDS07] . They model 
a workfiow as meeting a utility goal if it satisfies a temporal logic formula. Minimizing the amount 
of information disclosed is similar to an agent maximizing his reward and thereby not performing 
actions that have costs but no benefits. However, in addition to having different research goals, we 



18 



consider several factors that these works do not, including quantitative purposes that are satisfied 
to varying degrees and probabilistic behavior resulting in actions being for a purpose despite the 
purpose not being achieved. 



Expressing Privacy Policies with Purpose. Work on understanding the components of pri- 
vacy policies has shown that purpose is a common component of privacy rules (e.g., [BA051[BA08j ). 
Some languages for specifying access-control policies allow the purpose of an action to partially de- 
termine if access is granted |PS03l ICra02l IBKKF051 IBKK06j . However, these languages do not 
give a formal semantics to the purposes. Instead they rely upon the system using the policy to 
determine whether an action is for a purpose or not. 



Philosophical Foundations. Taylor provides a detailed explanation of the importance of plan- 
ning to the meaning of purpose, but does not provide any formalism |Tay66| . 

The sense in which the word "purpose" is used in privacy policies is also related to the ideas of 
desire, motivation, and intention discussed in works of philosophy (e.g., [Ans57j ). The most closely 
related to our work is that of Bratman's on intentions from which we get our model of sequential 
planning |Bra87j . In his work, an intention is an action an agent plans to take where the plan 
is formed while attempting to maximize the satisfaction of the agent's desires; Bratman's desires 
correspond to our purposes. Roy formalized Bratman's work using logics and game theory |Roy08| . 
However, these works are concerned with when an action is rational rather than determining the 
purposes behind the action. 

We borrow the notion of non-redundancy from Mackie's work on formalizing causality using 
counterfactual reasoning jMac74j . In particular, Mackie defines a cause to be a non-redundant part 
of a sufficient explanation of an effect. Roughly speaking, we replace the causes with actions and 
the effect with a purpose. The extension to our semantics proposed in Section 17. 2^ may be seen 
as another instance of non-redundancy. This time, we replace the causes with purposes and the 
effect with an action. This suggests that for an action to be for a purpose, we expect both that the 
action was non-redundant for improving that purpose and that the purpose was non-redundant in 
motivating the action. That is, we expect planning to be parsimonious. 



Planning. Psychological studies have produced models of human thought (e.g., ABB"*" 04] ). How- 



ever, these are too low-level and incomplete for our needs |DKP96j . The GOMS formalism provides 
a higher level model, but is limited to selecting behavior using simple planning approaches |JK96j . 
Simon's approach of bounded rationality |Sim55j and related heuristic-based approaches |GS02] 
model more complex planning, but with less precise predictions. 



9 Conclusions and Future Work 

We use planning to present the first formal semantics for determining when a sequence of actions 
is for a purpose. In particular, our semantics uses an MDP-like model for planning, which allows 
us to automate auditing for both restrictive and prohibitive purpose requirements. Furthermore, 
our semantics highlights that an action can be for a purpose even if that purpose is never achieved, 
a point present in philosophical works on the subject (e.g., |Tay66| ), but whose ramifications on 
policy enforcement had been unexplored. Lastly, our framework allows us to explain and compare 
previous methods of policy enforcement in terms of a formal semantics. 
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However, we recognize the limitations of this modeh it imperfectly models human planning and 
only captures some forms of planning for multiple purposes. Nevertheless, we believe the essence 
of our work is correct: an action is for a purpose if the actor selects to perform that action while 
planning for the purpose. Future work will instantiate our semantic framework with more complete 
models of human planning. 

Fundamentally, our work shows the difficulties of enforcement due to issues such as the tenable 
deniability of ulterior motives. These difficulties justify policies prohibiting conflicts of interest 
and requiring the separation of duties despite possibly causing inefficiencies. For example, many 
hospitals would err on the side of caution and disallow referral from a physician to his own private 
practice or require a second opinion to do so, thereby restraining the ulterior motive of profit. 
Indeed, despite the maxim that privacy is security with a purpose, due to these difficulties, purpose 
possibly plays the role of guidance in crafting more operational internal policies that organizations 
enforce rather than the role of a direct input to the formal auditing process itself. In light of 
this possibility, one may view our work as a way to judge the quality of these operational policies 
relevant to the intent of the purpose requirements found in the actual privacy policy. 

We further believe that our formalism may aid organizations in designing their processes to avoid 
the possibility of or to increase the detectability of policy violations. For example, the organization 
can decrease violations by aligning employee incentives with the allowed purposes. 

Acknowledgments. We appreciate the discussions we have had with Lorrie Faith Cranor and 
Joseph Y. Halpern on this work. We thank Dilsun Kaynar and Divya Sharma for many helpful 
comments on this paper. 
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A Details of MDPs 



One may find a discussion of MDPs in most introductions to artificial intelligence (e.g., |RN03j ) . 
For an MDP m = {Q,A,t,r,^), the discount factor 7 accounts for the preference of people for 
receiving rewards sooner than later. It may be thought of as similar to inflation. We require that 
7 < 1 to ensure that the expected total discounted reward is bounded. 
The value of a state q under a strategy a is 



V^{a,q)=E 



^7V(gi,(T(gi)) 



,i=0 



The Bellman equation shows that 

Vm{(T,q) = r{q,a{q))+^Yl ^il^^ilW) * Hn(f^,9') 

q'eQ 

A strategy a* is optimal if and only if for all states q, Vm{cr*,q) = maXg- Vm{cr,q). At least one 
optimal policy always exists. Furthermore, if a* is optimal, then 



cr*(q) = argmax 



q'&Q 



B Proof of Theorem [T] 

The proper sub-execution relation is a strict partial order. This follows directly from the proper- 
subsequence relation IZ being a strict partial order. We write for proper sub-execution and < for 
proper sub-execution or equal. 

Now, we show that -< is also strict partial ordering. 

• Irreflexivity: for no a is a ~< a. For cr ^ cr to be true, there would have to exist a o" G opt such 
that for at least one contingency k' and q' , m{q' , k' ,a') is a proper sub-execution of itself. 
However, this is impossible since the sub-execution relation is strict partial order. 

• Asymmetry: for all ai and (T2, if fxi -< (T2, then it is not the case that (72 -< a\. To show a 
contradiction, suppose a\ -< 02 and -< are both true. It would have to be the case that 
for all contingencies k and states g, m{q,K,ai) < m{q,K,a2) and m{q,K,a2) ^ m{q, K,ai). 
Since is a strict partial order, this implies that for all q and k, m{q, K,ai) = m(g, k, (T2). 
Thus, there cannot exist a contingency k' and state q' such that m{q' , k' , a2) < m{q' , k' ,ai). 
Then (T2 -< ox cannot be true, a contradiction. 

• Transitivity: for all a\, 02, and (T3, if ui -< 02 and (72 ^ (73, then a\ ^ (73. Suppose (J\ -< 02 
and (72 -< (73. Then for all for all contingencies k and states q, m(q, K,ai) < m(g, k, (72) and 
m{q,K,a2) ^ k, (73). Since < has transitivity, this implies that m{q,K,ai) < m{q,K,a^) 
for all K and q. 

Furthermore, it must be the case that there exists a contingency k' and state q' such that 
m{q' , k' , (7i) <m{q' , k' , (72). From above, 'm{q' , k' , 02) ^ ^{q' , k' , (73). Thus, by the transitivity 
of <, m{q' , k' ,(Ti) <i m(g', k', (73) as needed. 
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Since ^ is a strict partial ordering and Q ^ ^ is finite, Q ^ Ais well-founded under -<. Q ^ A 
being finite also means that opt(m) is finite. It is also known to be non-empty |RN03j . 

Suppose opt*(m) were empty. This would mean for every a of opt(m), there exists a' in opt(m) 
such that a' ~< a. Since opt(m) is finite but non-empty, this could only happen if -< contained 
cycles. However, this is a contradiction since ^ is a strict partial order and Q ^ A is well-founded 
under it. Thus, opt*(m) is not empty. 



C Proofs about Useless States 



Proposition 1. For all environment models m, sets U such that U Q Um, strategies a, and states 
q, Vm{(J,q) < Vm{U{a),q). 

Proof. Let exec(6) be all the executions with the behavior 5 as a prefix. Let Bu be the set of all 
behaviors b such that for some j, b = [qo,ai,qi . . . ,qj,aj^i,qj^i] such that {qj,aj+i) is in U but 
for not i < j is {qi-i, Oi) in U . We may use Bu and exec(6) to partition the space of executions E. 
Thus, 



^l'r{qi,a{qi)) 



Li=0 



J]Pr[e|a] 

eS-B 



i=0 



(1) 



b£Bu eeexec(6) 



^l'r{qi,a{qi)) 

oo 

^fr{qi,a{qi)) 



j=0 



(Note: as E is uncountable, taking a summation over it is ill advised. We could take an integral 
instead. Alternatively, one could take the sum over executions of bounded length. This will 
introduce an error term. However, as the bound increases the magnitude of this term will drop 
exponentially fast due to the factor 7. In essence, this is how most practical algorithms for solving 
MDPs operate. See [RN03].) 

For any b in Bjj, consider e € exec(6). Since e is in exec(6), it must have the following form 

[qo,ai,qi . . . ,qj,aj+i,qj+i, . . .] 

where (g^, a^+i) G U but for i < j is {qi, m+i) ^ U where b = [qo, ai, gi . . . , qj, aj+i, g^+i]. 
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For a G strg(6), we reason as shown as follows. 



e£exec(6) 



^l''r{qi,a{qi)) 



.1=0 



e€exec{6) 



i-1 



oo 



j=0 



i-1 



(2) 

Furthermore, 



eGexec(fe) i=0 

E Pr[e|a]E7V((?.,cT(g,)) 

eeexec(fe) i=0 

E Pr[e|a]E7V((?.,a(g,)) 

eGexec(fe) j=0 



E Pr[e|a]£y-V(g„a(g,)) 

e€exec(ti) 



i=j 



+ 7% 



j-1 i-1 
E Pr[e|a]E7V((?„a(g,)) =Pr[6|a]E7Vfe,cT((?.)) 

e6exec(fe) i=0 i=0 

Thus, the left term is equal under a and U{(t): 



7-1 



(3) 
(4) 
(5) 



E PT[e\a]Y,l'r{qi,a{q,))=FT[b\a]Y,fr{Qi,<^(.Q^)) 

eGexec(b) i=0 i=0 

= Pr[6|a]E7Vfe,C/(a)(g,)) 



E Pr[e|^(a)]E7V((Z.,C/(a)fe)) 

eGexec(b) 



j=0 



where line [H follows since (7{qi) = U{a){qi) for ((/j,aj+i) ^ C/. 

Since {qj, fflj+i) G C/, we know that Qm{o', qj,aj+i) < 0. Furthermore, since a € strg(6), it is the 
case that (7{qj) = Oj+i. Thus, 14n(c, Q'j) = Qm{cr,qj,a{qj)) < 0. Furthermore, since {qj,aj+i) € [/, 
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Vm{U{a),qj) = Qm{(^,Qj, N) = 0. Thus, we may conclude 

" oo 



J2 p^t^i^] 

eg exec (6) 



,i=0 



(6) 

(7) 

(8) 
(9) 
(10) 



i-1 



e£exec(fe) 



1=0 



i-1 



Fr[e\U{a)]Yyr{q,,U{a){q,)) 

eGexec(b) 



< 



i=0 



i-1 



Y Pr[e\U{a)]YlHQ^,U{a{qi))) 

eGexec(b) 



+ l^Vmia,qj) 

+ l^Vr,^{a,qj) 
+ j^Vm{U{a),qj] 



i=0 



e£exec(6) 



E7V(g.,C/(a)(g,)) 



j=0 



where hues [6] and [10] come from the reasoning leading to line [2l and line [7] comes from the reasoning 
leading to line O 

Note that the above also trivially holds when a ^ strg(6) since Pr[e|iT] = and Pr[e|[/((7)] = 
for all e G exec (6). Thus, for all a, we have 



(11) E 

e€exec(6) 



i=0 



eGexec(b) 



X;7V((?.,C/(a)(q.)) 



i=0 



Thus, 



(12) 



(13) 
(14) 



b^Bij eeexec(6) 

bGBu eGexec(6) 
= Kn,(f/(a),g) 



E7V(gi,o-(gi)) 

oo 

E7*?'(gi,o-(9i)) 



.1=0 



where line [12] and [2] comes from the reasoning of line [T] and line [13] comes from equation [TT] □ 

D Proof of Lemma [1] 

First we prove that this log^^(6)nbehv*(m) in the lemma may be replaced with strgj„(6) nopt*(?n). 
Then, we prove the modified statement with two propositions. We have one corresponding to the 
if direction and one to the only if direction. 

Proposition 2. For environment models m, if for all observable behaviors b, log(6) = b, then 
strg(6) nopt*(m) is empty if and only iflog~^{b) n behv*(?n) is empty. 
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Proof. Since log" (6) = {6}, log" (6)nbehv*(m) is empty if and only b ^ behv*(m). b is in behv*(m) 
if and only if there exists a strategy a in opt*(m) such that there exists a contingency k, and a 
state q such that 6 is a subsequence of m{q, k, a). 

For all a in opt*(m), 3K,q.b C m{q,K,a) is equivalent to Vi G [0,n).a{qi) = Oi+i where b = 
[qo,ai,qi,a2, ■ ■ ■ ,an,qn]- To see this, note b was observed and, thus, it must have been produced 
by a contingency consistent with m. 

Mi G [0, n).cr(g'j) = Oj+i is equivalent to a € strg(6). Thus, b is in behv*(m) if and only if there 
exists a strategy a in opt*(m) such that a is in strg(6). Thus, log~^(6) fl behv*(m) is not empty if 
and only if strg(6) n opt*(?7i) is not empty. □ 

Proposition 3. For all environment models m and behaviors b = [qo^ai^qi^ . . . ,amqn 

], strg(6) n 

opt*(m) is not empty if (1) for all i such that < i < n, (gj,aj+i) ^ Um and (2) strgib) n opt(m) 
is not empty. 

Proof. Suppose the conditions (1) and (2) are true. Since strg(6) nopt(m) is not empty, there exists 
some o"! in both of them. Since di is in strg(6), for all < i < n, (Ti((/i) = Oi+i- Thus, by condition 
(2), {qi,cri{qi)) ^ Um- This further implies that Oj+i is not N. 

Let (72 = Um{(Ti)- <72 is in strg(6) because for all < i < n, cri{qi) = (72{qi) since {qi,(7i{qi)) ^ 
Um- Furthermore, by Proposition [H for all q, Vm{(7i,q) < Vm{a2,q)- Thus, (72 is in opt(m) as well. 

To show that (72 is also in opt*(m), suppose it were not. Since (72 is in opt(m), this im- 
plies that there exists a' in opt(m) such that a' -< 02- For this to be true, there must exist k! 
and state g' such that active(m((7', k', cr')) IZ active(m(g', (72). Thus, for some i, m(g',K',(72) 
must have the form [(70, ai, f?!, Oi, ft, Oi+i, ft+i,-- •], and m{q',K',a') must have the form 

[(/o, ai, (/I, . . . , qi-i,ai, qi, N, qi, . . .] where Oj+i is not N. Since (T2{qi) = Cj+i, by the construction of 
1725 ('ZijOi+i) is not in Um- Thus, there exists some (73 such that (5m,((73, (/j, Oj+i) > 0. 

Since (72 is in opt(m), Qm(o"2, gi, Qi+i) > <3m(o-3, a^+i) > 0. Thus, we have Vm{(y2,qi) = 
Qm{o'2,Qi, Oi+i) > 0. However, Vm{cr', qi) = meaning that a' is not in opt(m), a contradiction. □ 

Proposition 4. For all environment models m and behaviors b = [qo, ai, qi, . . . , a„, i/strg(6) n 
opt*(m) is not empty, then (1) for all i such that < i < n, {qi, Oi+i) ^ Un and (2) strg(6) nopt(m) 
is not empty. 

Proof. Condition (2) follows from the fact that opt*(m) C opt(m). 

To prove condition (1), suppose strg(6) n opt*(m) is not empty but condition (1) does not 
hold. Then there exists ai in strg(6) fl opt*(m). Furthermore, there exists some i' such that 
{qii,aii^i) G Um- Since ai G strg(6), it must be the case that for all i < n, aj+i = cr{qi). Thus, 
f^iiQi') = Q^i'+i- By Proposition [H for all q, Vmicrijq) < Vm{Um{cri) , q) ■ Furthermore, U{ai) -< ai. 
To see this, recall that Um is not empty. Thus, any contingency k' that results in state qii, 
m{qQ, k' ,Um{cri)) IZ m{qo, k' ,ai) since only Umio'i) does nothing at qi/. For k that do not lead to 
qi', the two executions will be the same. 

Since Umio'i) -< ci and U{ai) is in opt(?7i), ai cannot be in opt*(&), a contradiction. □ 

E Proof of Lemma [2] 

If {q,a) is in Um, then a 7^ N and for all strategies a, Qm{cr,q,a) < 0. Thus, the lemma is true if 
the following is true: Q*{q, a) < iff Ma.Qm{cr, q, a) < 0. 
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To show this, note that ya.Qm{o', q,a) < iff maxo- Qmi'^, Q, o) < 0. Furthermore, 
maxQm(cr,g,a) = maxr{q,a) + 'yS^ t{q,a){q) * Vm{a,q') 

= '^(^jCi) +7y]*(9,«)(9') * maxKi(o-,g') 
= r(g,a)+7j^t(g,a)((?')*^*('7') 

Thus, V(T.Q„((T, g, a) < iff Q^X^l. a) < 0. 



F Properties of fix 

Proposition 5. For all environment models m, strategies a, and states q, Vfjx(m,fe) (c") q) ^ ^(c, q). 
Proof. Let m= {Q, A,t,r,^) andfix(m,6) = {Q,A,t,r','y). 



(15) 

(16) 
(17) 



Vm{a,q)=E 



^fr{qi,a{qi)) 



< E 



,i=0 
oo 



^fr'{qi,a{qi)) 
L't=o 

Vfix(m,fe)(o'i 



where hne 1161 follows from the fact that for all q and a, r'{q, a) < r{q, a). 



□ 



Proposition 6. For all environment models m, behaviors b, a £ strg(6), and states q, Vfjx{.m,6) (i^i q) = 
Vm{<7,q) 

Proof. Let m = {Q,A,t,r,-f) and fix(m, 6) = {Q,A,t,r',j). Let b = [qo,ai,qi, . . . ,an,qn]- 

Since a is in strg(6), for all i such that < i < n, a{qi) = flj+i. Thus, r'(gj,aj+i) = r(gj,aj+i). 
For all q that is not equal to qi for any i, r'{q, a) = r{q, a) for all a. Thus, for all a and q, 
''''{qi^iq)) = ^{q-i^^iq))- This implies 



Vm{a,q)=¥. 



E 



^l'r{qi,a{qi)) 



.1=0 
oo 



Y.^'r'{q.,a{q,)) 
Lj=o 

Vfix(m,b)(0') 



□ 



Proposition 7. For all environment models m, behaviors b, and ai ^ strg(6), there exists a 
0-2 € strg(6) such that for all states q, Vfix(m,b) (o"!, g) < Vfix(m,b) (0-2, g). 
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Proof. Let fix(m, 6) = {Q,A,t,r',^). Let b = [qQ,ai,qi, . . . ,an,qn]- Since cJi is not in strg(6), 
there must exist some i such that cri{qi) ^ Oi+i- Let the set / hold all such indexes i: I = 
{i € [0, n) I cri{qi) ^ Oj+i }. Let C72 be the strategy such that a2{q) = flj+i ii q = qi for some i £ I 
and 0"2((?) = cri{q) otherwise. By construction, CJ2 is in strg(6). 

By the construction of fix(m,6), for all i G I, r'{qi,ai{qi)) = —uj < r'{qi,ai+i) = r'{qi,a2{qi))- 
Thus, for all q, r'{q,ai{q)) < r'{q,a2{q)). Thus, for all states q, Vfix(„,^fe) (cti, g) < Vf;^(^rn,b){(^2, q)- □ 

Proposition 8. For all environment models m, behaviors b, cti ^ strg(6), and a2 € strg(6), there 
exists a state q such that Vfix(m,fe) (<7i j ^) < ^fix(m.,fe) (<^2 1 • 

Proof. Let b = [q^, ai,qi, . . . , a„, qn]. Since ai is not in strg(6), there must exist some i such that 
f^iiQi) 7^ CLi+1- By the construction of fix(m,6), r'{qi,ai{qi)) = —u). Recall that u) > 2r*/(l — 7) 
where r* is the reward with the largest magnitude. Thus, 

(18) ^fix(m,f))(o-i>%) =r{qi,ai{qi)) +jJ2^i(li^^^ili))i(l') *^rn{(T,q') 

q' 

(19) = + 7 t{qi, ai{qi)){q') * V^{a, q') 

q' 

(20) < _^ + ^ ^ t{qi, ai{qi))iq') * r*/{l - 7) 

q' 

(21) = -w + 7*r7(l -7) 

(22) <-w + r7(l-7) 

(23) <_[2r-7(l-7)]+r7(l-7) 

(24) =-r*/il-j) 

(25) <K^(^T2,g) 

(26) =yf\x{m,b){(^2,q) 

where line 12 II follows from t{q, (Ti{q) being a probability distribution over states, line 1251 follows from 
the definition of r* and known bounds (e.g., |RN03j ). and line [26] follows from by Proposition [6l □ 

Proposition 9. For all environment models m and behaviors h, opt(fix(m, 6)) is a subset o/strg(6). 

Proof. Suppose cJi were not in strg(6). By Proposition El for all a2 € strg(6), there exists a state q 
such that Vf jx(m,,fe) (o"! , <?) < Vf\y,(^jn,b)i^2,Q)- Thus, (Ji is not in opt(fix(m, 6)). □ 

Proposition 10. For all environment models m, behaviors b, and strategies a in opt(fix(m, 6)), 
Vfix(m,fe) (o-, 9) = Vm{cr,q)- 

Proof. Let a be in opt(fix(?7T,, 6)). a must be in strg(m, 6) by Proposition [9l Thus, Vfix(m,fe) (^^"5 ?) = 
Vm{(T,q) by Proposition [6l □ 

G Proof of Lemma [3] 

This lemma follows directly from the Propositions [11] and [12] below. 

Proposition 11. For all environment models m and behaviors b, strg(6) nopt(m) = opt(fix(m, 6))n 
opt(m). 
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Proof. Consider the set strg-opt(m, 6) = strg(6) — opt(fix(m, 5)). For all a in strg-opt(m, 6), a is 
in strg(6) but not opt(fix(m, 6)). By being in strg(6), Vfjx(m,b) ("^i = ^m{cr,q) by Proposition [H 
Thus, since a is not in opt(fix(m, 6)), a is not in opt(m) either by Proposition [5j This means that 
strg-opt(?n, b) D opt(?n) is empty. 

Furthermore, opt(fix(m, 6)) C strg(6) by Proposition [9l Thus, strg(6) = opt(fix(m, 6))U(strg(6) — 
opt(fix(m, 6))) = opt(fix(m, 6)) U strg-opt(m, 6). Thus, 

strg(6) nopt(m) = (opt(fix(m, 6)) U strg-opt(m, 6)) nopt(m) 

= (opt(fix(?n, b)) n opt(m)) U (strg-opt(m, b) n opt(m)) 
= opt(fix(m, b)) n opt(m) 

□ 

Proposition 12. For all environment models m and behaviors b, opt(m) fl opt(fix(m, 6)) is empty 
if and only if for all q, max^ Vf\x{m,b){(^, q) / max^ Vm{(7, q). 

Proof. Suppose that opt(m) opt(fix(m, 6)) is not empty. Then there exists a* in both of them. 
Thus, 



(27) max Vf ix(m,b) (cr, q) = Vfix(m,;,) (o"* , 

(28) =Vrn{a*,q) 

(29) = maxVmicr,q) 

a 



where line [28] follows from Proposition [TOl and lines 1271 and 1291 follow from a* being in both opt(m) 
and opt(fix(m, b)). 

Suppose that for all q, maxo- Vm(o', go) = ^^^aVf\x{m,b)i'^T Qo)- Let a* be in opt(fix(m, 6)). For 
all q, 



(30) Vrn{(T*,q) = Vfix(m,6)(cr*, g) 

(31) = max Vfix(m,f,) {a, qo) 

(32) = maxVmicr,q) 



where line 1301 follows from Proposition [10] and line [31] from a* € opt(fix(m, 6)). Thus, a* is in 
opt(m) and opt(?n) n opt(fix(m, 6)) is not empty. □ 

H Proof of Theorem [2] 

Line 05 will return true if there exists i such that Oj+i ^ N and Q^{qi,ai^i) < 0. By Lemma [21 
this implies that {qi,ai+i is in Um- By Lemma [TJ this implies that log~^(6) PI behv*(m) is empty 
under condition (1). 

Lines 06-16 constructs m' = fix(m, 6). It constructs r' from r by first setting r' = r. On lines 
13-16, it then sets r'{qi,k) to be —uj for all k such that k ^ Oj+i. Thus, r'(gj,aj+i) will be left as 
r((?j,aj+i) as needed. 
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If Line 05 does not Line 21 will return false if there exists j such that V^{j) = V^{j). In 
this case, it cannot be that for all q, maxo- V^((T, go) = ™aXo- Vfjx(m,b)('75 (Zo)- Thus, by Lemma El 
strg(6) n opt(m) is not empty and condition (2) is false of Lemma [TJ Since the function would had 
returned already at Line 05 if condition (1) were true, we know it is false. Thus, by Lemma [H 
log~^(6) n behv*(m) is not empty. 

If Line 22 is reached, true is returned. This can only happen if condition (2) is true. This 
implies that log~^(6) R behv*(m) is empty by Lemma [TJ 

Thus, the algorithm is correct whether it returns true or false. 
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